Perhaps in the era of AI we should not be surprised that one of the most successful stock market strategies of the 21st century is just an algorithm grabbing a basket of stocks by size. However I never cease to be amazed that a passive index can and does reliably outperform swathes of the investment management industry; at the time of writing explicitly passively managed assets reached over US $15 trillion, exceeding actively managed assets for the first time across asset classes. Increasingly even managed funds increasingly ‘hug the index’.
Two inventions underpin the rise of passive investing: Stock market indices and exchange traded funds (ETF). In this first article of a two part series, I explore the first half of the 20th century to understand why indexes exist and how that informs their design.
The roots of the modern index can be traced to the late 19th century, when Charles Dow—the co-founder of Dow Jones & Company—sought to simplify the interpretation of stock market performance for The Wall Street Journal’s readership. In 1884, Dow began publishing a simple average of selected railroad stocks in his Wall Street Journal, nee Customers’ Afternoon Letter. By 1896, this work had evolved into what became known as the Dow Jones Industrial Average (DJIA), which tracked a basket of 12 industrial companies, growing to 30 by 1926.
By selecting a handful of large, influential U.S. companies and computing their average share prices, Dow created a simple one-number metric through which investors could gauge the broader market’s direction. Investors' appetite for the stock market at the end of the 19th century was low as it was perceived as a highly speculative activity. However, journalists recognised that a concise figure could sell newspapers and attract traders; other financial news outlets soon followed suit. By the 1920s, indices became daily fixtures in the U.S. financial press. Newspapers such as the New York Herald Tribune began publishing their own versions, competing on the frequency of index data reported as well as the number of sectors and stocks included. In Europe the innovation caught on slower, with the Financial Times' (The Financial News at the time) FT30 index launched in 1935.
Although journalism and commercial interests sparked the earliest U.S. indices, an equally powerful driver emerged from within the budding field of macroeconomics. Beginning in the 1910s and 1920s, economists increasingly recognized that stock prices could act as leading indicators of economic cycles—especially if they were measured consistently over time.
One of the most important intellectual forces behind this new perspective was Wesley C. Mitchell, a pioneer of business cycle research. As a founding figure at the National Bureau of Economic Research (NBER), Mitchell posited that examining fluctuations in stock prices could help predict peaks and troughs in broader economic activity.
Mitchell and his colleagues needed extensive historical price data, which went beyond the short-term focus of daily newspapers. They required large samples of firms and long time spans—often spanning decades—to identify patterns and turning points with statistical rigor. This demand sparked the refinement or development of indices that accounted for mergers, capital reorganizations, and dividends. The crude methods of summing up share prices, typically used in news columns, gave way to more refined approaches that recognized a company’s evolving capital structure.
Early business cycle theorists saw indices as more than just snapshots: they were potential forecasting tools as leading indicators of the macroeconomy. By comparing key economic aggregates (industrial production, employment, freight shipments, etc.) with stock indices over time, early researchers found partial support for the hypothesis that stock prices often turned downward before the broader economy. That recessions and index downturns are often but not always related can be seen on the graph of the Dow-Jones above, where the grey bars represent recessions. Thus, the drive for better macroeconomic models and forecasts forced economists to build or refine indices that could withstand the scrutiny of long-term business cycle analysis.
The methodological and conceptual groundwork laid by Mitchell in 1916, as well as contemporaries such as J. Commons, N. Stone, and W. Persons, outlived their studies on macro cycles. By arguing that researchers must look beyond a handful of headline stocks, more broadly represent the sectoral composition of the market, and in particular adjust data for stock splits, dividends, and capital reorganizations, Mitchell laid the groundwork for the creation of the modern index.
In 1957 this was exemplified by the S&P 500 Stock Composite Index by Standard and Poors. Ultimately its timing and introduction was enabled by the concurrent computer revolution, with an electronic calculation method developed by Boston-based Melpar, Inc., that allowed S&P to perform index calculations much more efficiently than before, producing hourly results.
At their heart stock market indexes are simple creatures. They seek to answer the question: given a bundle of stocks what is their overall position. The generic formula for any index can be represented as:
$$\text{Index Value}_{t} = \sum_{n=1}^{N} \Bigl(\text{weight}_{t}^{n} \times \text{price}_{t}^{n}\Bigr)$$
Where the weighting function determines what kind of index we have created, e.g. Market Cap, Price, etc. “Weight” in most official index methodologies is not literally multiplied by the stock’s price to get an additive term; many index providers (e.g., S&P Dow Jones Indices) compute an index by summing float-adjusted market caps across constituents and then dividing by a “divisor” which the index continuous. Symbolically,
$$\text{Index} \;=\; \frac{\displaystyle \sum_{n=1}^{N} \Bigl(\text{float-adjusted shares}_{t}^{n} \times \text{price}_{t}^{n}\Bigr)}{\text{divisor}}$$
The “weight” is implicitly:
$$\text{weight}_{t}^{n} \;=\; \frac{\text{float-adjusted shares}_{t}^{n} \times \text{price}_{t}^{n}}{\displaystyle \sum_{k=1}^{N} \Bigl(\text{float-adjusted shares}_{t}^{k} \times \text{price}_{t}^{k}\Bigr)}$$
At this point it is important to understand the distinction between calculation and rebalancing:
This latter point becomes very relevant for index funds. If you are tracking (replicating) a market-cap index in a real portfolio, you might choose to fully replicate those floating weights only once a quarter (at rebalancing). Between rebalances, you typically let your holdings “drift” with the market. For funds there is pressure to rebalance less frequently in order to keep costs down. This will be covered more extensively in part 2.
From the index providers perspective, the index calculation is done effectively continuously. The “float shares” piece is only changed when corporate actions happen or at scheduled rebalancings. The price component moves continuously, so the implied “weight” of each constituent changes continuously.
Dearest reader, you've now seen where the indices came from, but wouldn't you like to make one yourself? The first step is the most important: why do you want to do this? Index construction begins with setting strategic objectives, before designing a rules-based approach to select for securities that deliver these objectives in the market. Ultimately your index’s overarching objective—be it benchmarking a local stock market, comparing companies within a single industry, or offering a tradable instrument for profit—will guide every subsequent design choice.
Index construction begins with setting strategic objectives, before designing a rules-based approach to select for securities that deliver these objectives in the market
Identify whether the index focuses on a single country, a region (e.g., Europe, Asia-Pacific), or has a global remit. A U.S.-only benchmark has a narrower lens than a global ex-U.S. index, for instance.
Consider which industries or economic segments the index should encompass. Should it be broad-based (covering all major sectors) or tailored to specific themes (e.g., renewable energy, healthcare)?
By nature, indices are passive, governed by transparent rules that automate security selection.
These filters ensure the index is both practical and representative:
Apply additional rules to refine the index constituents:
Once you know which securities enter the index, you must decide how much each one influences overall performance.
Type a stock name and press Enter to add it or type a number in “or randomly add” and press Enter to randomly add stocks. Click on any stock line to remove it. "Top K" adjusts how many stocks are included by order of size, "Rebalance Freqency" updates the periodicity of inclusion/exclusion, and "Weight Cap (%)" limits the maximum contribution any given stock can have on the index.